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Overviews, or syntheses of research syntheses, have become a popular 
approach to synthesizing the rapidly expanding body of research and system¬ 
atic reviews. Despite their popularity, few guidelines exist and the state of the 
field in education is unclear. The purpose of this study is to describe the 
prevalence and current state of overviews of education research and to pro¬ 
vide further guidance for conducting overviews and advance the evolution of 
overview methods. A comprehensive search across multiple online databases 
and gray literature repositories yielded 25 total education-related over¬ 
views. Our analysis revealed that many commonly reported aspects of sys¬ 
tematic reviews, such as the search, screen, and coding procedures, were 
regularly unreported. Only a handful of overview authors discussed the syn¬ 
thesis technique and few authors acknowledged the overlap of included sys¬ 
tematic reviews. Suggestions and preliminary guidelines for improving the 
rigor and utility of overviews are provided. 
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Research synthesis, a rigorous approach to cumulate evidence, has become an 
important technique to manage, integrate, and summarize the burgeoning research 
industry (Cooper, 2010). Researchers use syntheses to generate new knowledge and 
identify gaps in extant literature (Lipsey & Wilson, 2001; Pigott, 2012). Policymakers 
and practitioners increasingly rely on systematic reviews to inform funding alloca¬ 
tions and practice (Gough, Oliver, & Thomas, 2012). The research synthesis industry 
is efficient and expanding, nearly doubling each year in the social sciences (Williams, 
2012). Bastian, Glasziou, and Chalmers (2010) estimated that 11 systematic reviews 
are published daily in the online database MEDLINE alone. 

Largely as a result of the increase in systematic reviews, researchers have 
begun to synthesize the syntheses. This method of research synthesis (Becker & 
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Oxman, 2008), where the review is the unit being synthesized rather than the 
primary study, offers another means to precis the ever-increasing amount of 
research generated (Bastian, Glasziou, & Chalmers, 2010). These syntheses can 
produce answers to unique and important questions that other research methods 
cannot (Cooper & Koenka, 2012) and are often robust to sample or scale varia¬ 
tions, resulting in utility and practicality for policymakers and practitioners 
above and beyond systematic reviews. This method has been referred to by dif¬ 
ferent terms, including meta-meta-analysis (Hattie, 2009; Kazrin, Durac, & 
Agteros, 1979), meta-synthesis (Cobb, Lehmann, Newman-Gonchar, & Alwell, 
2009), overview (Pieper, Antoine, Morfeld, Mathes, & Eikermann, 2014a), 
overview of reviews (Cooper & Koenka, 2012), review of reviews (Maag, 
2006), second-order meta-analysis (Tamim, Bernard, Borokhovski, Abrami, & 
Schmid, 2011), tertiary review (Torgerson, 2007), and umbrella review 
(Thomson, Russell, Becker, Klassen, & Hartling, 2010). 

We adopt the terminology used by the Cochrane Collaboration and refer to a 
synthesis of reviews as an overview (Becker & Oxman, 2008). Overviews are 
becoming increasingly common in health sciences (Pieper, Buechter, Jerinic, & 
Eikermann, 2012; Thomson et al., 2010), and overviews’ results are extending 
into other areas including the social and education sciences (Cooper & Koenka, 
2012). As such, overviews have the potential to shape education policy and pro¬ 
vide guidance to researchers and practitioners alike. 

Examples of influential overviews are easily identified in the literature. For 
example, Higgins, Xiao, and Katsipataki (2012) synthesized 45 systematic 
reviews on the effects of digital technology on children’s learning. The authors 
provided a comprehensive summary of the findings as well as clear and direct 
recommendations to practitioners based on the totality of the studies. The over¬ 
view provided clarity to the discrepant systematic review results, making them 
easier to interpret and implement in practice. Consequently, Higgins et al.’s over¬ 
view has already been cited 15 times since it was published. Torgerson’s (2007) 
overview, cited 27 times since publication, combined 14 systematic reviews on 
the effects of literacy training. The reviews were grouped into content areas where 
specific conclusions could be drawn about each of the varying programs or inter¬ 
vention styles. The authors suggested that, based on the overview findings, spe¬ 
cific literacy training may be more appropriate for differing groups of students or 
intervention styles, a finding that would not be possible with a traditional or sys¬ 
tematic review that focuses on one or small set of studies. Finally, the largest 
education research overview conducted to date (Hattie, 2009) synthesized over 
800 reviews related to academic achievement and has been cited 113 times since 
2009. The overview synthesized well over 10,000 primary studies, and therefore, 
its conclusions may be more robust to sample and intervention variation. 
Moreover, the overview is able to provide comparisons across the reviews and 
thus make suggestions and inferences not possible using individual reviews. 

Although overviews are becoming prevalent and may offer advantages over 
traditional research syntheses, overviews are a relatively nascent and undevel¬ 
oped synthesis method that pose unique methodological challenges (Cooper & 
Koenka, 2012; Thomson et al., 2010) and may be problematic (Hartling, 
Chisholm, Thomson, & Dryden, 2012; Pieper et al., 2012). It is unclear to what 
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extent overviews are being conducted in education research, the methods used to 
conduct education overviews and synthesize results, or how valid this research 
method is. The purpose of this study, therefore, is to examine the prevalence of 
education overviews, assess the current state of overviews in education, outline 
the unique challenges and contributions overviews may provide above and beyond 
systematic reviews, and provide preliminary guidelines to education researchers 
based on the review’s results. 

Unique Contributions From Overviews 

Overviews can make unique contributions to the knowledge base above and 
beyond systematic reviews and be advantageous to policymakers, practitioners, 
and researchers alike. Overviews can provide a broader summary of evidence for 
use by stakeholders and researchers (Cooper & Koenka, 2012) and can be used to 
examine trends and changes in research over time. Overviews allow for the 
research problem to be defined in a broader way, capture a variety of interventions 
being used to treat similar conditions, or be used to identify variation in the types 
of outcomes, problems, populations, or contexts of the same intervention (Becker 
& Oxman, 2008; Cooper & Koenka, 2012). 

Another advantage of the overview is the ability to compare and contrast 
results across multiple systematic reviews. The rapid growth of the systematic 
review industry means that reviews on the same topic will occur, and those 
reviews may result in varying conclusions. It might be difficult to discern concor¬ 
dance or discordance among reviews without the context of an overview. One 
illustrative example is from the literature on school bullying prevention programs. 
Merrell, Gueldner, Ross, and Isava (2008) synthesized 15 studies across 15 differ¬ 
ent outcomes. The results for the reduction in bullying perpetration indicated only 
a small, nonsignificant intervention effect, k=S,d = .04. On the other hand, Ttofi 
and Farrington (2011) synthesized 89 studies of 53 evaluations divided into only 
two main constructs, bullying perpetration and bullying victimization. The aver¬ 
age results for the reduction in bullying perpetration indicated a larger and statisti¬ 
cally significant treatment effect, k = 41, d = .17. By synthesizing these two 
reviews, overview authors have the opportunity to compare and contrast the varia¬ 
tion between these discordant reviews based on differences in research questions, 
populations, methods, or other characteristics (Pieper et ah, 2012). As such, iden¬ 
tifying and examining discrepancies and agreements across reviews provide valu¬ 
able evidence that could be used by stakeholders and researchers to advance 
scientific knowledge and practice. 

A third advantage offered by overviews is the ability to conduct a network 
meta-analysis (Ioannidis, 2009). A network meta-analysis is applicable when 
multiple interventions and control groups are compared. This analysis allows the 
researcher to understand differences across interventions or comparisons, even 
when direct comparisons were not made within the reviews. A relevant yet hypo¬ 
thetical example derives from literature on the impact of various interventions to 
increase math test scores. One systematic review collects studies that tests cur¬ 
riculum changes, another synthesizes the effects of teacher professional develop¬ 
ment, and a third review includes studies that examine the effectiveness of 
curriculum changes compared with teacher professional development and both of 
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those types to a control group. Using a network meta-analysis, one is able to com¬ 
pare across each of the various combinations of interventions in addition to the 
simple comparison of intervention versus control. Cooper and Koenka (2012) 
described one such scenario in the context of medical research. To date, however, 
researchers have not attempted an analysis of this kind in education, but it is likely 
a logical next step. 

A final advantage to conducting an overview is that it elucidates when system¬ 
atic reviews need an update. The Campbell Collaboration (2014), the leading pro¬ 
ducer of systematic reviews in the social sciences, suggests that reviews be 
updated at least every 3 to 4 years. The Cochrane Collaboration, the leading pro¬ 
ducer of systematic reviews in the medical sciences, suggests an update may be 
necessary even sooner (Higgins & Green, 2008). An overview that synthesizes the 
corpus of systematic reviews will recognize if an update to a specific field is 
required. Pieper, Antoine, Neugebauer, and Eikermann (2014c) provided a helpful 
framework to assess whether a particular systematic review is up to date, which 
could be used across reviews. 

Taken together, overviews can be useful and enlightening to inform policy, 
practice, and research. The use of overviews, however, relies on their validity, 
applicability, and methodological rigor. As such, the community must maintain 
high standards for such overviews, similar to the way that methodologists have 
argued for higher standards in systematic reviews (Moher, Liberati, Tetzlaff, & 
Altman, 2009). 

Conducting an Overview 

The conduct and organization of an overview, in many ways, is very similar to 
a systematic review. Cooper and Koenka (2012) suggested that an overview mir¬ 
rors the steps of a systematic review, following the suggestions of Cooper (2010) 
or Lipsey and Wilson (2001). As illustrated in Table 1, the parallels between the 
two methods are striking, and overview researchers, in the face of few guidelines, 
would do well to simply follow the methodological suggestions of systematic 
reviewers. The need to formulate a well-conceived research question, search the 
literature for relevant studies, extract data from studies, evaluate the studies, ana¬ 
lyze and integrate the outcomes, and interpret and present the evidence are the 
basis of rigorous synthesis methods (Cooper, 2010). Although the major steps of 
conducting an overview are analogous to conducting a systematic review, impor¬ 
tant differences remain within these steps that are critical to consider. 

One of the most significant differences between conducting a systematic 
review compared with an overview is the need for overview authors to consider 
multiple study levels—the overview level, the review level, and the primary study 
level—throughout the process and take steps to minimize bias and error at all 
levels. Overview authors may introduce bias and error through their methodologi¬ 
cal procedures and by including reviews that contain bias and errors. Authors 
introduce bias through their own methods and through the inclusion of possibly 
biased primary studies. Indeed, overview authors compound bias and error when 
the methods used at the overview, review, and primary study levels are not evalu¬ 
ated. Overview authors therefore must consider not only how they conduct the 
overview but also how the review authors conducted their review. 
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Subgroup analyses Use moderator or meta-regression analyses No prescribed guidelines; consider using moderator or meta¬ 
following set guidelines (Cooper et al., 2009) regression techniques 

Publication bias Reported on population of studies Report results from the individual reviews and consider 

analyzing the overviews 
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The importance of taking both the overview and the review level into 
account during the overview process is first apparent when determining eligi¬ 
bility criteria. Overview authors must explicate eligibility criteria related to 
their primary unit of analysis—the review—in addition to the eligibility crite¬ 
ria they define for the primary studies included in the reviews. The eligibility 
criteria, for example, may specify that the overview will include only system¬ 
atic reviews (i.e., descriptive reviews are excluded) that examined effects of an 
intervention using randomized controlled trials only. In this case, overview 
authors must attend to both the design of the reviews (systematic reviews only) 
as well as the study designs included within the reviews (randomized con¬ 
trolled trials only). In this example, any systematic review that includes pri¬ 
mary studies other than randomized-controlled trials would therefore be 
ineligible for inclusion. 

Overview data extraction takes a similar form to coding primary studies for a 
systematic review, but the information extracted is often quite different. The dif¬ 
ference again lies in the multiple levels embedded within the overview process. 
For an overview, the author extracts data related to the primary unit of analysis— 
the included reviews. Overview authors must also consider what data they will 
extract related to the primary studies included in the reviews, and they can choose 
to include or ignore reporting on primary studies. Ignoring the primary studies, 
however, likely results in an incomplete portrayal of systematic review findings 
and thus the credibility of the overview would be questionable. An illustrative 
example is study design. A systematic review that includes many types of con¬ 
trolled and uncontrolled studies differs from a systematic review that only includes 
randomized controlled trials. The average effect sizes may be similar across the 
two reviews, but the internal validity of each primary study differs greatly. 
Therefore, it is important that overview authors code and report pertinent infor¬ 
mation about the systematic review as well as information the systematic review 
reports about the primary studies. 

Study quality is another component of the overview process that must be con¬ 
sidered at the review and primary study levels. For systematic reviewers, it is 
critical to assess study quality and risk of bias of included studies because prob¬ 
lems with the design and execution of primary studies have implications for the 
inferences gleaned from the review (Valentine & Cooper, 2008). Overview 
authors must also consider the quality of the included reviews as well as the qual¬ 
ity of the primary studies constituting the reviews. High-quality systematic 
reviews may include many or mostly low-quality primary studies. The validity of 
the conclusions drawn across included systematic reviews relies on the quality of 
the overview, reviews, and primary studies. 

The final component of the overview process to consider is in the synthesis of 
the results. Similar to a systematic review author, the overview author can elect to 
describe each study individually, conduct a descriptive synthesis, or quantitatively 
synthesize the results of the reviews using meta-analytic techniques. The criti¬ 
cisms of descriptive and vote counting methods of synthesizing outcomes (see 
Cooper, 2010) apply equally to overviews. In terms of a quantitative synthesis of 
results, overview authors are faced with more complexity than review authors. 
Overview authors may choose to extract and synthesize primary study level effect 
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sizes (where available) and variances or they may choose to extract the average 
effect size calculated and reported by review authors. If the overview authors 
choose to quantitatively synthesize mean effects across reviews, little guidance is 
available; however, Schmidt and Oh (2013) described methods for second-order 
meta-analysis when each meta-analysis reports the results from a random-effects 
model. 


Current State of Overview Methods 

Increase in the demand and production of research syntheses engenders the 
need for more credible methods of synthesizing evidence (Cooper, 2010). As a 
result, the methods of research synthesis have been advancing dramatically over 
the past 20 years. Multiple research articles, books, and journals devoted to 
research synthesis methods have been published to improve the practice of 
research synthesis and advance the science of research synthesis methods (Shadish 
& Lecy, 2015). Although significant empirical work has been undertaken to 
inform and improve research synthesis methods to minimize bias and error in the 
review process and increase the credibility and validity of review findings 
(Cooper, 2010; Moher etal., 2009; What Works Clearinghouse, 2015), limited 
research or guidance is available to the overview author. 

The Cochrane Collaboration endorses overviews and has published guidance 
on the conduct and reporting of health-related overviews (Becker & Oxman, 
2008). Cooper and Koenka (2012) and Thomson et al. (2010) offered a descrip¬ 
tion of the steps and methods overview researchers have adapted from other meth¬ 
ods and the challenges inherent in the overview process. The What Works 
Clearinghouse’s (2015) guidelines explicitly discuss reviewing education-related 
topics, but focus exclusively on reviewing primary studies. Limited extant empiri¬ 
cal inquiry in education regarding overview methods is available, however, and 
the limited empirical work published in this field is primarily in medicine and 
health (Thomson et al., 2013). 

Pieper and colleagues (Pieper et al., 2012,2014a; Pieper, Antoine, Neugebauer, 
& Eikermann, 2014c) published a series of studies examining the rigor, overlap, 
and up-to-dateness of overviews in the health sciences. They found in their review 
of 126 overviews that there was much heterogeneity in the conduct of overviews, 
and many overviews lacked methodological rigor. Moreover, only about half of 
the overviews considered overlap of reviews, with the possibility of certain pri¬ 
mary studies being included more than once, which gives disproportionate statis¬ 
tical power to those primary studies (Pieper etal., 2014c). Up-to-dateness is 
another characteristic of overviews that has been examined empirically, with find¬ 
ings pointing to overview authors’ lack of attention to whether the reviews are 
providing the most up-to-date evidence (Pieper etal., 2014a). Thomson etal.’s 
(2013) review of 29 overviews concluded that considerable work is still needed 
on the methods of overview research. 

Cooper and Koenka (2012) and others (Pieper et al., 2012, 2014c; Thomson 
et al., 2013) have called for researchers to examine and advance overview meth¬ 
ods. Despite these calls, however, overview methods in education research have 
largely been overlooked. The purpose of this study, therefore, is to build on prior 
studies to further elucidate overview methods and expand this research into the 
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field of education. By examining the extant overviews of education research, we 
can describe the prevalence and current state of overviews in education, compare 
the overview methods used by education researchers to that of health sciences, 
and begin to provide further guidance to conducting overviews of education 
research and advance the evolution of overview methods. The research questions 
guiding this study are the following: (a) To what extent are overviews being con¬ 
ducted in the area of education related research with preschool to postsecondary 
student populations? (b) To what extent are methodological characteristics being 
reported in overviews of education? (c) What methods are overview authors using 
to conduct overviews? We conclude by suggesting preliminary overview method¬ 
ological guidelines based on the answers to these questions. 

Method 

Systematic review procedures were employed to search, select, and extract 
data from overviews that meet eligibility criteria for this study. We followed the 
Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) 
guidelines where applicable (Moher et al., 2009). 

Eligibility Criteria 

Eligible overviews must have aimed to synthesize more than one empirical 
education-related review (e.g., narrative, systematic review, or meta-analysis) 
with preschool, primary, secondary, or postsecondary students, including special 
or general populations. For the purposes of this study, we considered the overview 
focused on an education related topic if the overview authors explicitly reported 
that the focus was on education in the title or the abstract, or if at least 50% of the 
included reviews synthesized effects of school-based interventions or education- 
related outcomes. We did not restrict our search to any time frame, and we 
searched for both published and unpublished reports; however, we included only 
English language reports. Relevant authors were contacted to inquire about poten¬ 
tial missing studies. 

Search Procedures 

Seven electronic databases were searched in September 2014 to identify eligi¬ 
ble overviews: Academic Search Premier, Education Complete, ERIC, ProQuest 
Dissertations and Theses, PsychlNFO, Science Direct, and Social Sciences Citation 
Index. Keyword searches within each electronic database included variations of 
the following keyword terms: “meta-review,” “umbrella review,” “review of 
review,” “overview of review,” “meta-meta-analysis,” “overview,” “meta-analysis 
of meta-analyses,” “synthesis of review,” and “synthesis of systematic review.” 
Hand-searching of reference lists and forward citation searching using Google 
Scholar were conducted with articles identified during the search process as well 
as with the following articles identified prior to the search: Cooper and Koenka 
(2012), Thomson et al. (2010, 2013), Pieper, Buechter, Jerinic, and Eikermann 
(2012), Pieper, Antoine, Morfeld, Mathes, and Eikermann (2014a, 2014c). We 
searched the gray literature using Google Scholar. The full search strategy for each 
electronic database is available in Appendix A (available in the online version of 
the journal). 
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Study Selection and Data Extraction 

Titles and abstracts of the overviews found through the search procedures were 
screened for relevance by one author. If the report appeared to be eligible, or if 
there was any question as to the appropriateness of the report at this stage, the full 
text document was obtained and independently screened by two authors using a 
screening instrument to determine inclusion. Any discrepancies between authors 
were discussed and resolved through consensus, and when needed, a third author 
reviewed the study. 

Studies that met inclusion criteria were coded using an author-developed data 
extraction instrument comprising the following sections: (a) bibliographic infor¬ 
mation; (b) overview characteristics and methods; (c) information overview 
authors provided about included reviews; (d) overlap, quality, and up-to-dateness 
of included reviews; (e) overview synthesis methods; and (f) Google Scholar cita¬ 
tion rates. The data extraction instrument, available from the authors, was pilot 
tested with five of the included overviews by two authors and adjustments to the 
coding form were made. Two authors then independently coded the remaining 
overviews using a FileMaker Pro database (Apple, Inc., 2014). Initial interrater 
agreement was 92.5%. Discrepancies between the two coders were discussed and 
resolved through consensus, and when needed, a third reviewer was consulted. 

Analytic Procedures 

The studies were analyzed descriptively for purposes of reporting. We aimed 
to elucidate all aspects of the overviews including where the overviews failed 
to collectively report important methodological details. We calculated the per¬ 
centage of characteristics reported across different methodological aspects. We 
also elected to test for differences in the reported methodological characteris¬ 
tics across time, and we hypothesized that overviews published more recently 
would report more information. To investigate whether reporting of method¬ 
ological characteristics varied across time, we selected 8 of the 17 coded char¬ 
acteristics to be summed (i.e., databases reported, keywords reported, time 
frame reported, abstract screening process, full-text screening process, coding 
procedures reported, gray literature searched, eligibility criteria reported) 
within each overview. We choose these eight characteristics because (a) the 
characteristics could be used in each study (i.e., all overviews searched online 
databases but not all overviews conducted a quantitative synthesis) and (b) the 
characteristics could be easily reported in the overview. A Pearson product- 
moment correlation was estimated between the summation and the time vari¬ 
able. We also split the sample into recent and early overviews and calculated a 
t test. We created all figures using Microsoft EXCEL and conducted analyses 
using base R (R Core Team, 2015). 


Results 

Search Results 

Titles and abstracts of the 6,566 citations retrieved from electronic 
searches of bibliographic databases and additional citations reviewed from 
reference lists of prior reviews and forward citation searching were screened 
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FIGURE 1. Flow chart of search and selection process. 


for relevance. Of those, the full texts of 258 reports were retrieved and inde¬ 
pendently screened for inclusion by two authors. Twenty-five overviews 
reported in 27 reports met eligibility criteria for this study. The 231 reports 
deemed ineligible were excluded for the following reasons: not being an 
overview (n = 209), not being education related (n = 16), not focused on 
Pre-K through postsecondary populations (n = 2), or not published in English 
(n = 4). A list of excluded studies is available in Appendix B (available in the 
online version of the journal). See Figure 1 for a flow chart summarizing the 
search and selection process. 

Descriptive Characteristics of Included Overviews 

The majority of included overviews were conducted with the purpose of exam¬ 
ining effects of interventions (80%). Five authors reported conducting the over¬ 
view for other purposes: to examine effects of schooling, describe the status of 
knowledge in reading, explore relationships of various education-related factors, 
and to consolidate and examine the results of the meta-analyses compared with 
work of other researchers. The overviews covered a variety of topics, including 
learning and educational achievement, science education training, instructional 
systems and design, curriculum, literacy and reading, technology, social skills 
training, school-based health promotion, mental health and social/emotional 
learning and well-being, social skills training, special education interventions, 
and self-determination. Although some overviews were found in the gray litera¬ 
ture, most (80%) were published in peer-reviewed journals. Publication dates 
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ranged from 1983 to 2012 ( SD = 9.63). The overviews included a median of 30 
reviews (range 5 to 800) and no overviews included primary studies. Yearly cita¬ 
tions rates were high for the included overviews; the median overview received 
6.85 citations per year, and the top overview had been cited more than 1,800 times 
(Lipsey & Wilson, 1993). 

Methodological Characteristics of Overviews 

We investigated the reported methodological characteristics of overviews and 
identified 17 distinct methodological criteria: online database searching, hand 
searching, reference harvesting, author contacts, targeted website search, key¬ 
words identified, time frame searched and included, gray literature sources, 
abstract screening, full-text screening, coding procedures, eligibility criteria, syn¬ 
thesis technique, meta-analysis procedures, adjustments for varying meta-analy¬ 
sis techniques, subgroup analyses, and publication bias. Table 2 delineates many 
of these characteristics across each of the overviews. 

Notably, many aspects that systematic reviews routinely report were lacking in 
the overviews. Although the online databases were often reported (64%), other 
critical aspects of the search, such as reference harvesting (48%), author contact¬ 
ing (16%), and hand searching (40%), were reported less than half the time. Only 
about a third of the overviews reported any of the keywords used to search the 
online databases (36%), and 40% of overviews reported the allowable time frame 
of reviews. With regard to the eligibility criteria, about half (56%) of the over¬ 
views reported at least some criteria, but few studies (16%) reported every impor¬ 
tant eligibility criterion. In addition, few overviews detailed how they screened 
abstracts (24%) and full-text reports (28%), or how they extracted data from the 
reviews (28%). 

Reporting Trends Across Time 

Across the eight eligible categories, only two overviews (8%) reported all 
eight methodological characteristics (Tamim et al., 2011; Torgerson, 2007). 
Across the 25 overviews, an average of 3.56 (SD = 2.60) of the eight character¬ 
istics were reported. One possibility for the lack of methodological reporting is 
that a portion of the overviews was published prior to widespread acceptance 
and use of reporting standards and guidelines. We therefore sought to deter¬ 
mine the relationship between time and the reported methodological character¬ 
istics. Figure 2 illustrates the reporting trend across time. It is clear that 
overview authors have begun to report more methodological aspects of their 
studies, especially compared with those conducted in the 1980s and early 
1990s. The Pearson product-moment correlation between the number of 
reported methodological characteristics and the year is positive and statisti¬ 
cally significant (r=A\,p= .04). In addition, we also dichotomized the sam¬ 
ple into recent (2000-2015) and early (1983-1999) overviews. The number of 
reported methodological characteristics is greater in more recent overviews 
(M= 4.31, SD = 2.63) compared with earlier reviews (M = 2.75, SD = 2.42), 
but the difference is not statistically significant, t( 23) = 1.55, p = .13. 
Nevertheless, it is clear that methodological reporting has improved (d = .60), 
although there is still need for improvement. 
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B = book, G = government report); Reviews = number of included reviews; Search characteristics (DB = reported database search, HS = hand search, 
RE = reference harvesting, AU = contact authors, WS = targeted website, SE = search engine, KW = keywords, TF = time frame, GL = gray literature 
[1 = government report, 2 = private report, 3 = books, 4 = conference presentation, 5 = dissertations and theses, 6 = other]); Screen/Code characteristics 
(AS = abstract screening, FS = full-text screening, CO = coding process reported, EL = eligibility criteria [1 = population, 2 = setting, 3 = intervention/ 
relationship, 4 = design, 5 = outcome, 6 = other]). 




♦ Total Methods Reported -Linear (Total Methods Reported) 


FIGURE 2. Number of methodological characteristics of overviews reported over time. 


Synthesis Methods 

We also investigated the synthesis methods used by overview authors 
(Table 3). Of the 25 overviews included, 11 (44%) used a descriptive review 
technique, electing to summarize each review textually and without quantita¬ 
tive analysis, whereas 14 of the 25 overviews (56%) used a quantitative ana¬ 
lytic technique. Of these 14, four elected to average the review results using a 
simple nonweighted average (16%), and five chose to weight the results by the 
number of included reviews (20%). Five of the overviews did not report how 
they synthesized the results (20%). Of the overviews that conducted a quanti¬ 
tative analysis, only a small proportion conducted subgroup (16%) or publica¬ 
tion bias (12%) analyses. Notably, none of the studies that used a quantitative 
technique considered or adjusted for various meta-analytic models and none 
of the studies used the technique proposed by Schmidt and Oh (2013). Taken 
together, the methodological aspects of the meta-analyses lacked clear and 
consistent reporting. 

Reported Characteristics of Reviews Included in Overviews 

We also evaluated the reported characteristics of reviews included in the over¬ 
views (Table 4). We examined 13 distinct and important categories: publication 
type, number of included studies, time frame searched, databases searched, search 
and screen procedures, coding procedures, study designs included, study quality, 
outcome type, analysis procedures, average effects, moderator/sensitivity analy¬ 
ses, and publication bias. 

The results of coding these aspects revealed systematic deficiencies across 
the overviews. For example, only one overview (4%) reported the search and 
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TABLE 3 


Methods used to synthesize reviews in included overviews 


Method 

n 

Percentage 

Synthesis method 

Descriptive 

ii 

44 

Quantitative 

14 

56 

Quantitative 

N/A: Descriptive review 

11 

44 

Did not report 

5 

20 

Simple average 

4 

16 

Weighted by number of reviews 

5 

20 

Adjusted for varying models 

N/A: Descriptive review 

11 

44 

Yes 

0 

0 

No 

14 

56 

Subgroup analysis 

N/A: Descriptive review 

11 

44 

Yes 

4 

16 

No 

10 

40 

Publication bias analysis 

N/A: Descriptive review 

11 

44 

Yes 

3 

12 

No 

11 

44 


screening procedures used by the included reviews, meaning only one over¬ 
view identified how each included systematic review conducted their search 
and screening of included primary studies. Along the same theme, only two 
overviews identified the coding procedures of the included systematic reviews 
(8%). Also somewhat surprisingly, only two overviews reported the databases 
searched by each of the reviews (8%), and only one overview reported the time 
frames of included studies in the reviews (4%). Some other aspects of the 
reviews, however, were more consistently reported by the overview authors. 
Most overviews reported the number of studies included in each review (60%). 
A majority of studies reported the types of outcomes coded within each review 
(64%), the analysis procedures of the reviews (84%), and the average results 
(76%). 

We also examined whether overview authors limited their study to include 
only systematic reviews (as a means of controlling for quality), or assessed the 
quality of included reviews and, if so, what they used to assess quality. Of the 25 
overviews included in this review, six (24%) overview authors limited the inclu¬ 
sion of reviews to systematic reviews. Eight of the overviews assessed and 
reported review quality in some way. One overview author used the QUORUM 
(Torgerson, 2007), one used the Critical Appraisal Skills Program (Tennant et al., 
2007), and the remaining reviews used author-developed tools. 
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TABLE 4 

Review characteristics reported in the overviews 


Reported review characteristics in the overview 


Author (DoP) 

T 

NS 

Tf 

SS 

Db 

Co 

SD 

Qu 

Ou 

An 

AE 

M/S 

PB 

Anderson (1983) 









V 

V 

V 



Browne et al. (2004) 

V 

V 






V 

V 

V 

V 


V 

Cobb et al. (2009) 

V 

V 

V 




V 

V 

V 

V 



V 

Cook et al. (2008) 

V 

V 





V 

V 


V 

V 


V 

Diekstra (2008) 


V 







V 

V 




Fomess (2001) 


V 







V 

V 

V 



Fraser et al. (1987a) 














Fraser et al. (1987b) 














Fraser et al. (1987c) 









V 

V 

V 



Green et al. (2005) 


V 







V 

V 




Gresham (1998) 










V 

V 



Guthrie et al. (1983) 














Hattie (1992) 


V 







V 

V 

V 



Hattie (2009) 


V 








V 

V 



S. Higgins et al. (2012) 









V 

V 

V 

V 


Kulik and Kulik (1989) 


V 







V 

V 

V 



Lipsey and Wilson (1993) 


V 







V 

V 

V 



Lister-Sharp et al. (1999) 

V 

V 



V 


V 

V 

V 

V 

V 


V 

Maag (2006) 


V 





V 



V 

V 



Sipe and Curlette (1997) 

V 




V 

V 


V 

V 

V 

V 

V 

V 

Tamim et al. (2011) 


V 








V 

V 



Tennant et al. (2007) 







V 


V 

V 

V 



Torgerson (2007) 

V 

V 





V 

V 

V 

V 

V 


V 

Wang et al. (1993) 











V 



Weare and Nind (2011) 

V 

V 


V 


V 

V 

V 

V 

V 

V 

V 

V 

Percentage reported 

28 

60 

4 

4 

8 

8 

28 

28 

64 

84 

76 

12 

28 


Note. V = Reported information; Author = Author represents the reference; DoP = date of 
publication; T = publication type; NS = number of studies; TF = time frame; SS = search/screen 
strategy; DB = search databases; CO = coding strategy; SD = study design; Qu = study quality; Ou = 
outcome type; AN = analysis procedure; AE = average effect; M/S = moderator/sensitivity analyses; 
PB = publication bias; Purpose (1 = To assess the effects of effectiveness reviews, 2 = To review or 
measure quality or methodological issues, 3 = Other). 


Overlap and Up-to-Dateness of Reviews in Overviews 

In the majority of the overviews (68%), overview authors did not address the 
issue of overlap in any way, four of the authors acknowledged that reviews 
included some of the same primary studies, and three authors accounted for over¬ 
lap in some way (e.g., removed highly overlapping reviews from analysis). Only 
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one author provided a matrix of all primary studies included in each review, 
clearly identifying the primary studies that were included in multiple reviews. 

In terms of how current, or up-to-date, the overviews were, we assessed publi¬ 
cation lag by calculating the difference between the mean publication year of the 
included reviews and the publication year of the overview. We also calculated the 
proportion of reviews published more than 5 years prior to the overview (Pieper 
et al., 2014c). The mean publication lag was 7.67 years. Many of the overviews 
included a large proportion of older reviews. The proportion of included reviews 
published 5 or more years prior to the overview ranged from 0% to 100%, with a 
median of 69%. 


Discussion 

The purpose of the present study was to examine the extent that overviews are 
conducted in education, to assess the methodological characteristics of education 
overviews, and to provide recommendations to advance overview methods and 
reporting standards given the current state of the evidence. Our systematic and 
comprehensive search of the education literature yielded 25 overviews for inclu¬ 
sion. Overall, our findings suggest a serious lack of methodological reporting and 
use of rigorous methods for conducting overviews, even when considering the 
fact that a portion of the reviews were conducted prior to widespread adoption of 
reporting guidelines for primary studies and systematic reviews. Many overviews 
failed to provide specific details about the search, screen, and coding procedures 
and a large portion of the overviews did not report many aspects of the eligibility 
criteria. Overall, we identified three major concerns about overviews in education 
research: (a) lack of reporting of methods used and characteristics of included 
reviews and primary studies, (b) sparse attention to overlap across reviews, and 
(c) underreporting of procedures used to synthesize the reviews. Although dis¬ 
turbing, these results should not be surprising given the previously conducted 
studies of overviews’ findings in the health and medical fields (Hartling et al., 
2012; Pieper et al., 2012; Thomson et al., 2013) that found similarly lacking 
methodological rigor and reporting of key information. 

One of the most concerning results from the present study is related to the 
reporting by overview authors, both in terms of reporting the overview methodol¬ 
ogy used and reporting information about the included reviews and primary stud¬ 
ies. This omission is especially apparent with regard to the selection and coding 
of reviews, which are crucial to the validity of a review. The practice of omitting 
crucial study selection and coding details is akin to omitting how participants are 
selected and data collected for primary studies. Compared with results of similar 
items assessed in studies of overviews in other fields, fewer education overviews 
(24%) reported methods for study selection compared with 49% found in Hartling 
et al.’s (2012) and Li et al.’s (2012) reviews in health. A smaller proportion of 
education overviews (28%) were also found to report data extraction procedures 
compared with 60% in the Hartling et al. (2012) and 44% in the Li et al.’s (2012) 
medical review of overview study. As seen in Figure 2, more recent overviews 
reported more methodological characteristics than those conducted in prior 
decades, yet better reporting is still needed. 
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Although methodological characteristics are important to inform the quality 
and validity of overviews, reporting characteristics of the reviews and primary 
studies are also important to the quality, validity, and relevance of the overview. 
Of the overviews included in this study, overview authors provided insufficient 
information about the included reviews and rarely provided basic information 
about the primary studies. Most overview authors did not report quality indicators 
of the included reviews; however, a greater proportion of education overviews 
reported quality indicators (28%) compared with Li et al.’s (2012) review where 
only 7% of the overviews reported quality indicators, but fewer than what was 
found by Hartling et al.’s (2012) study where 36% reported quality indicators. 
When primary study information was present, it was usually in regard to the num¬ 
ber of included studies or average effect size. The ability to judge the validity of 
an overview rests in the methods and quality of the reviews and the primary stud¬ 
ies included in those reviews. A serious deficit of information regarding the 
included reviews and primary studies inhibits assessment of the validity and, ulti¬ 
mately, the relevance of an overview. 

Overlap of primary studies across included reviews within an overview is 
another area of concern and has implications for the validity of overviews. When 
conducting a primary study synthesis, it is well-established that two reports of the 
same study should not be included, as that would cause a duplication of data; 
review authors are well aware of the need to ensure independence of effect sizes. 
Overview authors must be aware of a similar problem when conducting an over¬ 
view, assess the amount of overlap between reviews, and handle overlap if prob¬ 
lematic. Similar to findings in health care overviews (Pieper et al., 2014a), most 
education overview authors did not assess or address the issue of overlap in any 
way, and only three accounted for the overlap they found. Cooper and Koenka 
(2012) summarized various strategies overview authors have used to handle over¬ 
lap, including selecting the review that is most rigorous, contains the most evi¬ 
dence, provides the most complete description, is the most recent, or is published 
in a peer-reviewed journal. Overview authors may also choose to disregard over¬ 
lap all together and include all reviews in the overview. It is not clear which, if 
any, is the most appropriate approach to handling overlap. Each approach may be 
justifiable depending on the overview, although none of the approaches are likely 
to be completely adequate (Cooper & Koenka, 2012). It is clear that the problem 
of overlap in overviews has not been well addressed and methodological work in 
this area is needed. 

In examining synthesis methods used in the overviews, about half of the over¬ 
views employed a descriptive synthesis method, providing a textual summary of 
the reviews and results from each included review. Summarizing the results of the 
included reviews was the primary focus, rather than on identifying and analyzing 
the discordance between reviews. Simply summarizing the results of each study 
is problematic in the same way traditional descriptive syntheses are problematic 
for synthesizing primary studies (Combs, Ketchen, Crook, & Roth, 2011). A 
descriptive overview should maintain a similar methodological rigor to quantita¬ 
tive overviews despite the lack of a quantitative synthesis. At the very least, 
descriptive overviews can provide a comprehensive portrait of the systematic 
reviews available. 
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About half of the overviews, on the other hand, quantitatively synthesized the 
included reviews in some way. This is far greater than the proportion of overviews 
in Hartling et al.’s (2012) study, where they found that only 3% of the overviews 
conducted a quantitative analysis. Of the overviews in the present study that con¬ 
ducted a quantitative synthesis, few reported the specific statistical procedures 
used to synthesize the results, none corrected for or commented on the diversity 
of meta-analytic models, and none used the statistical procedures for combining 
effect sizes across reviews suggested by Schmidt and Oh (2013). Moreover, few 
studies conducted sensitivity analyses, such as publication bias analyses to evalu¬ 
ate the validity of the population of reviews. Subgroup and moderator analyses 
were also used infrequently. Given the potential advantages to quantitatively syn¬ 
thesizing reviews, it is important that methods for synthesizing results of reviews 
be developed and tested. 

Although the number of education overviews to date is far fewer than the num¬ 
ber of overviews of health related research (Hartling et al., 2012; Li et al., 2012; 
Pieper et al., 2012), overviews of education research reflect data from hundreds of 
primary studies and represent hundreds of thousands of students. Indeed, the 
median number of reviews included in the overviews was 30, with each review 
including anywhere from 5 to 800 studies. Moreover, overviews are highly cited 
and as a result have the potential to affect policy, practice, and future research. 
Using Google Scholar as the source, the overviews in this study were cited a 
median of 88 times as of February 2015, with one overview cited more than 1,800 
times (Lipsey & Wilson, 1993). Given the potential impact of overviews on prac¬ 
tice and policy, it is essential that overviews are conducted in a rigorous way to 
minimize bias and error and provide the most valid results possible. Unfortunately, 
limited guidance is available to inform the conduct and reporting of overviews 
(Cooper & Koenka, 2012) and there is a need for more clear conduct and report¬ 
ing guidelines for overviews similar to those that have been developed for system¬ 
atic reviews (Hartling et al., 2012; Pieper et al., 2012, 2014a; Thomson et al., 
2010 ). 

Preliminary’ Conduct and Reporting Guidelines for Overviews 

Results from the present study revealed significant deficiencies in the conduct 
and reporting of education research overviews. To ensure the validity and utility of 
overviews to inform education practice and policy, it is important that the conduct 
and reporting of overviews improve. Because the nature of an overview follows a 
similar structure and tone as a systematic review, simply following the standards 
developed for systematic reviews will greatly improve future overviews. Although 
the conduct and reporting of overviews can be guided, in large part, by established 
conduct and reporting guidelines of systematic review methods, important differ¬ 
ences remain between a systematic review and an overview that require a distinct 
set of guidelines. Building on the recommendations made by Cooper and Koenka 
(2012) and the Cochrane Collaboration (Becker & Oxman, 2008) along with sev¬ 
eral other sources (Campbell Collaboration, 2014; Chandler, Churchill, Higgins, 
Lasserson, & Tovey, 2013; Moher et al., 2009; Pieper et al., 2012, 2014a; Pieper, 
Antoine, Neugebauer, & Eikermann, 2014b, 2014c; Smith, Devane, Begley, & 
Clarke, 2011; Thomson et al., 2010), we offer the following Preliminary Conduct 
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and Reporting Guidelines for Overviews. A summary of the Preliminary Conduct 
and Reporting Guidelines for Overviews can be found in Table 5. 

Title and Abstract 

The reporting standards for systematic reviews related to titles and abstracts 
can be applied to overviews. As specified in the PRISMA (Moher et ah, 2009) 
and Meta-Analysis Reporting Standards (MARS; American Psychological 
Association, 2010) reporting standards, the title should identify the type of 
study being reported, specifically the title should identify the report as a system¬ 
atic review or a meta-analysis. Along the same lines, we recommend that the 
title of an overview also clearly identify the type of study being reported. A 
number of different terms currently exist to identify the type of study we refer 
to as an overview; however, we encourage the field to be consistent in the ter¬ 
minology and adopt the term overview of reviews, or more simply overview, 
used by Cochrane and several others conducting research in this area (e.g., 
Becker & Oxman, 2008; Cooper & Koenka, 2012; Pieper et ah, 2014a; Thomson 
et ah, 2013). 

Also similar to PRISMA and MARS reporting standards, we recommend that 
abstracts for overviews use a structured format and provide a summary of the key 
components of the study: background and purpose; method (including eligibility 
criteria, data sources, synthesis method); results (including sample size, charac¬ 
teristics of included reviews and primary studies, quality assessment); and con¬ 
clusions (including implications and limitations). Although none of the 25 
included overviews used a structured format, Weare and Nind (2011) explicitly 
stated each of the key components in their abstract. 

Introduction and Research Questions 

The structure of the introduction for a report of an overview is also very similar 
to any other report of an empirical study. The introduction should provide a sum¬ 
mary of the problem under study, why the problem is important, and discussion of 
prior research and theory related to the problem under investigation. The intro¬ 
duction of an overview, however, differs from reports of primary studies and sys¬ 
tematic reviews and meta-analyses in that the introduction should provide a strong 
rationale for the need and appropriateness of synthesizing multiple reviews as 
opposed to conducting a first-order review by synthesizing the primary studies. 
The introduction should also include a clear statement of the research questions, 
and if appropriate, research hypotheses. Ideally, if the overview is addressing a 
question related to effectiveness of interventions, the research question should 
follow the PICOS format, specifying the population, intervention, comparison 
condition, outcomes, and study design. Of the overviews included in the present 
study, Torgerson’s (2007) review on literacy learning in English provides a good 
example of summarizing the literature appropriately and stating specific research 
objectives. 

Overview Methods 

Standards for the conduct and reporting of data sources and search procedures 
can be wholly adopted from systematic review conduct and reporting standards. 
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Title N/A Identify the report as an overview, preferably identifying 

whether descriptive or quantitative synthesis methods 
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primary studies. 

Search Develop specific search procedures for each source, preferably Report in detail all search procedures for each source 

procedures with assistance from an information retrieval specialist. searched in such a way that the search could be repeated. 

This can be provided in supplementary material. 



TABLE 5 (continued) 
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dateness date. Consider including recent primary studies that were overview, 

conducted after the included review timeframes. 
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Additional nuance that overview authors should consider include the need to add 
information sources and search terms to capture reviews rather than primary stud¬ 
ies. Overview authors should also consider contacting both review and primary 
study authors. 


Search and study selection. The search for overviews should follow familiar 
guidelines present in any systematic review report. Major databases should be 
searched thoroughly and systematically and the authors would do well to track 
the quantity and type of overviews retrieved from each. Although publication 
bias may or may not be an issue in terms of a review being published, it is nev¬ 
ertheless important to search the gray literature. A thorough search of Google 
Scholar is one place to start, but conference abstracts and relevant research firms 
are also informative. Should the overview author intend to include primary stud¬ 
ies in addition to reviews, these searches should be tailored to retrieve both types 
of studies. Finally, the overview author should consult a librarian or information 
retrieval specialist when planning the search. 

Study selection procedures for overviews are similar to procedures for select¬ 
ing primary studies for a systematic review. We recommend using at least two 
independent reviewers at each stage of the selection process, with transparent 
reporting of these procedures and decisions at each stage of the selection process. 
It is important when determining eligibility criteria for study selection that over¬ 
view authors take care to determine study design criteria at the review level as 
well as the primary study level. Considering the type of review design (limiting to 
only systematic reviews and how this is defined) or limiting reviews that include 
only randomized controlled trials or other study designs are examples of review 
and methodological characteristics that will need to be considered when setting 
eligibility criteria for an overview. For instance, Diekstra’s (2008) overview on 
school-based social and emotional education programs included in the present 
study provided a thorough and clear description of the eligibility criteria. 

Data collection. It is standard practice in a systematic review and meta-analysis 
to use a predetermined coding document and two independent coders. The recom¬ 
mendation for an overview should follow suit, and the overview author should 
consider conducting regular meetings with the coders to ensure coder drift does 
not occur. In terms of specific information collected from the overview, we suggest 
that overview authors report information collected by the review authors and char¬ 
acteristics of primary studies included in the reviews. The reviews contain crucial 
information that affects the validity of the overviews. Including low-quality or 
biased reviews relegates the overview to lower quality and biases the results of the 
overview. Audiences must be able to ascertain aspects of the population, and in this 
case, the reviews are the population. Without such information, it will be difficult 
to discern differences across overviews accurately. Lister-Sharp et al. (1999), for 
example, carefully articulated the coding and data extraction procedures. 
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Assessment of Methodological Quality 

The quality and validity of an overview is dependent on the quality of the 
included reviews and the primary studies included in those reviews. Thus, it is 
crucial that overview authors assess the quality of the reviews and the primary 
studies included in those reviews. This is a much more complex task than that 
faced by systematic review authors. Overview authors should describe the meth¬ 
ods used for assessing the quality of the included reviews and the evidence that is 
included in those reviews. Several tools are available for assessing methodologi¬ 
cal quality of reviews (e.g., Assessing the Methodological Quality of Systematic 
Reviews; Shea et al., 2007) and primary study evidence (e.g., Grading of 
Recommendation, Assessment, Development, and Evaluation; Guyatt et al., 
2008). The newly created Risk of Bias in Systematic Reviews tool is also now 
available as an option (Whiting et al., 2016). The tool is geared toward medical 
research, and as such some items may not be appropriate or applicable to educa¬ 
tion and social science. Another option is to create a tool specific to the topic area 
using the guidelines proposed in Cooper (2010) or Lipsey and Wilson (2001). 
Cooper’s (2010) text is especially appropriate, and Table 8.1 (p. 222) is an excel¬ 
lent guide. Given the lack of research on quality assessment of systematic reviews, 
we will not recommend any one tool for evaluating review or primary study qual¬ 
ity or risk of bias; however, it is strongly recommended that the overview authors 
clearly report their method for assessing methodological quality of included 
reviews and primary studies included in those reviews and provide rationale for 
the methods they used. 


Overlap. The degree of overlap is a methodological quality issue that needs to be 
addressed when conducting an overview. Including several reviews that have a high 
level of overlap could give disproportionate weight to one or a small number of 
reviews, and thus could bias the results of the overview and lead to erroneous con¬ 
clusions. Pieper et al. (2014c) described two ways of assessing overlap that overview 
authors could consider: calculating the “covered area” or the “corrected covered area” 
(p. 370). Both of these methods use a citation matrix, with the latter method making 
some adjustments to reduce the influence of a single large review. 

Unfortunately, no clear guidance is available on how to best assess or mitigate 
overlap; however, it is of utmost importance that overview authors plan to assess 
and handle overlap a priori and examine and report the level of overlap across 
included reviews. When authors recognize a high level of overlap and choose to 
handle overlap in some way, it is important that the authors clearly report the 
methods used to handle overlap and the results of that approach (e.g., clearly 
identify overviews excluded due to overlap). From the set of included reviews 
included in this study, Lipsey and Wilson (1993) considered the overlap of pri¬ 
mary studies and attempted to ameliorate the issue. 

Up-to-dateness. The up-to-dateness of reviews is also important to consider when 
conducting an overview. Including outdated reviews with older primary studies 
may not be comparable in terms of relevance or quality of more current reviews 
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and primary studies, and may disregard more recent studies that have not yet 
been included in a review. Thus, overviews may be out of date and not reflective 
of the current state of the evidence. Reflective of the recommendations of Pieper 
et al. (2014c), we recommend that overview authors attend to the up-to-dateness 
of the overview. Minimally, authors can examine the age of the studies included 
in the reviews as well as the reviews themselves and report and discuss the up- 
to-dateness of the evidence. Calculating the publication lag is another strategy of 
assessing up-to-dateness (Pieper et al., 2014c). Furthermore, if authors find gaps 
in the inclusion of more recent evidence, overview authors are encouraged to 
search for and include recent primary studies in the overview. 

Synthesizing Results of Reviews 

Methods of synthesizing review results offer unique conduct and reporting 
challenges over synthesizing primary study results because there are more com¬ 
plexities and little guidance for synthesizing reviews. Nevertheless, there are 
some key advantages to synthesizing reviews, including the potential to make 
comparisons among interventions examined in different reviews and the opportu¬ 
nity to employ more sophisticated analyses to allow both direct and indirect com¬ 
parisons (Thomson et al., 2010). Cooper and Koenka (2012) identified three 
primary approaches to synthesizing evidence from reviews: examining discor¬ 
dance between reviews, performing second-order meta-analysis, or performing a 
new meta-analysis by including all of the primary studies that were included in 
the reviews. We argue, however, that the third option, including all of the primary 
studies included in the reviews, would then be a new review and not an overview, 
and we will thus not discuss that option here. Unfortunately, techniques for quali¬ 
tatively or quantitatively synthesizing reviews are in their infancy and must be 
further developed. For overview authors who are conducting a descriptive synthe¬ 
sis of reviews, we recommend minimally examining and describing the discor¬ 
dance of included reviews as suggested by Cooper and Koenka (2012). We 
strongly discourage overview authors from using a vote-counting method, where 
the authors simply identify the number of reviews that found overall positive 
effects, null effects, and negative effects. 

Although methods for quantitatively synthesizing mean effects across reviews 
are not well developed, overview authors may have good reason to quantitatively 
synthesize effects across reviews. When possible, we suggest that overview 
authors consider quantitatively synthesizing review results. Overview authors 
must consider, however, the statistical implications of combining average effect 
size estimates across multiple reviews. To date, only Schmidt and Oh (2013) have 
put forward procedures for quantitatively synthesizing results from meta-analysis 
using random effects models. Although random-effects meta-analytic models are 
becoming commonplace, fixed-effect models are still used (Polanin & Pigott, 
2014). Synthesizing random-effects results with fixed-effect results should be 
treated as cautionary and, at a minimum, discussed as a limitation. We prefer that 
overview authors not combine these two types of effect size estimates and instead 
contact study authors for the appropriate results. Alternately, an overview author 
could estimate the random-effects model using effect sizes reported in the reviews. 
In addition to techniques for quantitatively synthesizing reviews, overview 
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authors must consider overlap and take appropriate steps to handle primary study 
overlap. 

Limitations 

Although this study is the first to examine education research overviews and 
contributes to the sparse empirical research on this developing research synthesis 
method, the findings of this review must be interpreted in light of the study’s limi¬ 
tations. The search process, although comprehensive for the subject matter, was 
constrained to education-related topics. Fields outside of education may conduct 
superior overviews or regulate the reporting of overviews. This is unlikely, how¬ 
ever, and we hope to investigate overviews in other fields in the future. It is also 
possible that studies are published in languages other than English. We believe the 
likelihood that many additional overviews exist in other languages is low, but 
nevertheless we could have missed a few. Although we did not limit our search to 
the United States, our search yielded only four overviews published outside of the 
United States. The overviews, however, no doubt included reviews published out¬ 
side the United States. An additional limitation is our lack of overview quality 
rating; however, we did code for a variety of overview methodological character¬ 
istics that would likely constitute any measure of overview quality and did use the 
PRISMA checklist to guide our construction of the coding form. 

Finally, we are susceptible to the traditional limitations of systematic reviews. 
Our work should be critiqued as if it were a systematic review and is only as good 
as the methods we used to collect and synthesize the studies, although we 
attempted to use systematic review best practices in conducting and reporting the 
results. Moreover, we are aware of the level of abstraction that comes from dis¬ 
secting overviews, which are themselves reviews of reviews. We must be cautious 
when discussing the direct implications of these types of studies, while under¬ 
standing that researchers are conducting overviews and need guidance. 

Conclusion 

The overview offers an exciting, yet challenging method for synthesizing and 
managing the ever-expanding volume of education research. Overviews provide 
unique opportunities to answer more broad and different research questions than 
we can answer using primary research or research synthesis methods. The results 
of this study, however, revealed significant deficiencies in the reporting, conduct, 
and synthesis of overviews in education research. Thus, caution must be used in 
interpreting and using results of extant overviews of education research. This 
study also supports the need for further development of overview methods and 
quality assessment tools; it is important that empirical work on the methodology 
for conducting overviews be undertaken to advance this novel synthesis method 
and inform best practices. 

Although conduct and reporting guidelines for systematic reviews are now 
commonplace and required to be followed by some journals, there has been little 
guidance for the conduct and reporting of overviews. Due to the added complex¬ 
ity inherent in the multiple levels of an overview, systematic review guidelines are 
not adequate, and thus, we have offered Preliminary Guidelines for the Conduct 
and Reporting of Overviews. We hope that the development of overview methods, 
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particularly methods for quantitatively synthesizing reviews, will follow the rapid 
progress of systematic review and meta-analytic methods and that these prelimi¬ 
nary guidelines are further developed as advances are made. Advancing the sci¬ 
ence of overview methods will take concerted time and effort, which we believe 
is necessary given the increase in the use of overview methods and the potential 
of this method to answer important questions. 

Note 

Research for the current study was partially supported by an Institute of Education 
Sciences Postdoctoral Training Fellowship Grant to Vanderbilt University’s Peabody 
Research Institute (R305B100016). Opinions expressed herein do not necessarily reflect 
the opinions of the Institute of Education Sciences. 
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